<html>
<head><meta charset="utf-8"><title>How much Ram do we need? · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html">How much Ram do we need?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="235324614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235324614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235324614">(Apr 20 2021 at 11:07)</a>:</h4>
<p>I am again thinking that our ram consumption seems unreasonable</p>



<a name="235324703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235324703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235324703">(Apr 20 2021 at 11:07)</a>:</h4>
<p>In the rust-analyzer directory, I did <code>rm ./target -rf &amp;&amp; cargo check &amp;&amp; cargo vendor &amp;&amp; cat vendor/**.rs &gt; all.rs</code></p>



<a name="235324780"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235324780" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235324780">(Apr 20 2021 at 11:08)</a>:</h4>
<p>So, the size of all our vendored deps is 37 megs</p>



<a name="235324842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235324842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235324842">(Apr 20 2021 at 11:08)</a>:</h4>
<p>the size of the target directory after check is 500 megs</p>



<a name="235325007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235325007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235325007">(Apr 20 2021 at 11:10)</a>:</h4>
<p>I think the size of the target should represent the <em>upper</em> bound on rust-analyzer's ram consumption -- <code>cargo check</code> processes everything, while we ignore function bodies (which should be the majority of code).</p>



<a name="235325059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235325059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235325059">(Apr 20 2021 at 11:10)</a>:</h4>
<p>Yet, rust-analyzer occupies about 1GB of ram</p>



<a name="235326973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235326973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235326973">(Apr 20 2021 at 11:28)</a>:</h4>
<p>The size of <code>~/.cache/JetBrains</code> after cleaning and working with rust-analyzer a bit is 150 megs</p>



<a name="235327217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235327217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235327217">(Apr 20 2021 at 11:31)</a>:</h4>
<p>So, either all those on-disk stores do compression, and it is 10x efficient (which is plausible) or we are using unreasonable amount of RAM for something?</p>



<a name="235329484"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235329484" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235329484">(Apr 20 2021 at 11:50)</a>:</h4>
<p>Finally stopped whining about comparing with CLion and actually compared with CLion: <a href="https://github.com/rust-analyzer/rust-analyzer/issues/7330">https://github.com/rust-analyzer/rust-analyzer/issues/7330</a></p>



<a name="235330391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235330391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235330391">(Apr 20 2021 at 11:57)</a>:</h4>
<p>interesting result! looks like there's still more data to intern/deduplicate then?</p>



<a name="235330450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235330450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235330450">(Apr 20 2021 at 11:58)</a>:</h4>
<p>ah, we still duplicate all the file contents, right?</p>



<a name="235330480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235330480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235330480">(Apr 20 2021 at 11:58)</a>:</h4>
<p>that should cost us around 50 MB</p>



<a name="235330645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235330645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235330645">(Apr 20 2021 at 11:59)</a>:</h4>
<p>We could claw back about 20 MB with <a href="https://github.com/rust-analyzer/rust-analyzer/issues/869#issuecomment-695171218">https://github.com/rust-analyzer/rust-analyzer/issues/869#issuecomment-695171218</a></p>



<a name="235331000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331000">(Apr 20 2021 at 12:02)</a>:</h4>
<p>Another interesting bit: clion's adress space compresses much worse than ours</p>



<a name="235331068"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331068" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331068">(Apr 20 2021 at 12:02)</a>:</h4>
<p>ideally, we wouldn't keep library code in memory at all, I guess? just a bit complicated to do with salsa</p>



<a name="235331286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331286">(Apr 20 2021 at 12:04)</a>:</h4>
<p>while we're at it, we could save the 'HIR' for libraries on disk and load it back, which would also solve the startup time problem and be useful for WASM/playground <span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span></p>



<a name="235331396"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331396" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331396">(Apr 20 2021 at 12:05)</a>:</h4>
<p>Uhu</p>



<a name="235331447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331447">(Apr 20 2021 at 12:05)</a>:</h4>
<p>And, do do that, we probably want to make sure that hir data is a flat array, which we can jusm memmap</p>



<a name="235331509"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331509" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331509">(Apr 20 2021 at 12:06)</a>:</h4>
<p>and that by itself might bring some wins</p>



<a name="235331532"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331532" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331532">(Apr 20 2021 at 12:06)</a>:</h4>
<p>Or dump it in SQLite :D</p>



<a name="235331654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331654">(Apr 20 2021 at 12:07)</a>:</h4>
<p>somewhat relatedly, I've also wondered whether we could load our HIR from rustdoc's json <span aria-label="grimacing" class="emoji emoji-1f62c" role="img" title="grimacing">:grimacing:</span></p>



<a name="235331692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235331692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235331692">(Apr 20 2021 at 12:08)</a>:</h4>
<p>I mean, making a normalized relation model out of hir is probably going to far, but using SQLite as a key-value store is probably on of the best way to do on-disk storage</p>



<a name="235332085"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235332085" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235332085">(Apr 20 2021 at 12:10)</a>:</h4>
<p>(unrelated, <a href="https://github.com/rust-analyzer/rust-analyzer/issues/8599">https://github.com/rust-analyzer/rust-analyzer/issues/8599</a> might be a nice benchmark for memory and loading times)</p>



<a name="235332367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235332367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235332367">(Apr 20 2021 at 12:12)</a>:</h4>
<p>I'm not sure we need something like sqlite when we just basically want to mmap a big datastructure and dump it fully back sometimes</p>



<a name="235332665"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235332665" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235332665">(Apr 20 2021 at 12:15)</a>:</h4>
<p>something like <a href="https://crates.io/crates/rkyv">https://crates.io/crates/rkyv</a> might be helpful? (somewhat new and experimental though)</p>



<a name="235333077"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333077" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333077">(Apr 20 2021 at 12:18)</a>:</h4>
<p>It probably doesn't matter that much, but SQLite is a pretty safe choice. Something like <code>sled</code> might be nicer when it matures, but you can't go wrong with SQLite</p>



<a name="235333431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333431">(Apr 20 2021 at 12:21)</a>:</h4>
<p>I would usually agree, but I think we need almost none of the features of sqlite (like indexing, or doing any kind of update except writing the full database)</p>



<a name="235333529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333529">(Apr 20 2021 at 12:21)</a>:</h4>
<p>it would also mean having a mandatory dependency on C code, which we currently try to avoid</p>



<a name="235333557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333557">(Apr 20 2021 at 12:21)</a>:</h4>
<p>Yeah,  but at least it's a way forward if we want/can move more stuff to disk</p>



<a name="235333701"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333701" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333701">(Apr 20 2021 at 12:22)</a>:</h4>
<p>I kind of want to do a prototype of this now <span aria-label="grimacing" class="emoji emoji-1f62c" role="img" title="grimacing">:grimacing:</span> maybe I'll find some time this week</p>



<a name="235333777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235333777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235333777">(Apr 20 2021 at 12:23)</a>:</h4>
<p>Like what if <code>salsa</code> could use an on-disk store? (It would probably be too slow for us)</p>



<a name="235334144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334144">(Apr 20 2021 at 12:26)</a>:</h4>
<p>this probably does need some salsa integration in any case, right?</p>



<a name="235334184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334184">(Apr 20 2021 at 12:26)</a>:</h4>
<p>we could also just use this <a href="https://github.com/michaelwoerister/odht">https://github.com/michaelwoerister/odht</a></p>



<a name="235334207"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334207" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334207">(Apr 20 2021 at 12:26)</a>:</h4>
<p>Yeah, not gonna happen too soon</p>



<a name="235334209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334209">(Apr 20 2021 at 12:26)</a>:</h4>
<p>not necessarily, I think. I think the general "salsa persistence" problem is quite a bit harder</p>



<a name="235334219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334219">(Apr 20 2021 at 12:26)</a>:</h4>
<p>it was written for use in rustc to store the incremental compilation cache</p>



<a name="235334461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334461">(Apr 20 2021 at 12:28)</a>:</h4>
<p>this could IMO be done by having some <code>if let Some(persistent_store) = db.persistent_store_for_crate(crate) { return persistent_store.get_data(id) }</code> in a bunch of queries</p>



<a name="235334614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235334614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235334614">(Apr 20 2021 at 12:29)</a>:</h4>
<p>with some hackery to make the <code>persistent_store_for_crate</code> query only be invalidated if the store file changes (or never, for a start)</p>



<a name="235335216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235335216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235335216">(Apr 20 2021 at 12:33)</a>:</h4>
<p>that aside, there still seems to be significant duplication that we can address without persistence</p>



<a name="235335376"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235335376" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235335376">(Apr 20 2021 at 12:34)</a>:</h4>
<p>unfortunately all the tooling for this sucks (case in point: you literally had to write the address space to disk and compress the file to get this information), so I'm not really sure how we're supposed to figure out <em>what</em> gets duplicated without auditing the entire codebase</p>



<a name="235336488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235336488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235336488">(Apr 20 2021 at 12:42)</a>:</h4>
<p>yeah... makes me wonder why there isn't better tooling to analyze memory dumps. With debug info, that should be possible in principle, right?</p>



<a name="235336601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235336601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235336601">(Apr 20 2021 at 12:43)</a>:</h4>
<p>Not unless you record a stack trace on allocation (like DHAT)</p>



<a name="235336621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235336621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235336621">(Apr 20 2021 at 12:43)</a>:</h4>
<p>DHAT even has all the info to help with this, it just doesn't, or it does but its UI is too convoluted</p>



<a name="235336727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235336727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235336727">(Apr 20 2021 at 12:44)</a>:</h4>
<p>DHAT computes stuff like "on average only N bytes of these M byte allocations get used"</p>



<a name="235336772"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235336772" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235336772">(Apr 20 2021 at 12:44)</a>:</h4>
<p>but not "hey this string is kept in memory 7 times" apparently</p>



<a name="235337818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235337818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235337818">(Apr 20 2021 at 12:51)</a>:</h4>
<blockquote>
<p>this could IMO be done by having some if let Some(persistent_store) = db.persistent_store_for_crate(crate) { return persistent_store.get_data(id) } in a bunch of queries</p>
</blockquote>
<p>I'd probably do this a bit differntly</p>



<a name="235338061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235338061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235338061">(Apr 20 2021 at 12:52)</a>:</h4>
<p>I think we can just change values to cache stuff on disk. Like, an Item tree can contain a <code>PathBuf</code> and load its data from disk on the first access (and evict data to disc under memory pressuer)</p>



<a name="235338205"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235338205" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235338205">(Apr 20 2021 at 12:53)</a>:</h4>
<p>That is, that'll allow us  to transparently unload some ram do disk. This won't help us with avoding computation during initial load.</p>



<a name="235338697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235338697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235338697">(Apr 20 2021 at 12:57)</a>:</h4>
<p>we don't really have a good way to compute memory pressure without a custom allocator</p>



<a name="235338739"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235338739" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235338739">(Apr 20 2021 at 12:57)</a>:</h4>
<p>maybe this can be hooked into salsa GC if the API for that is extended</p>



<a name="235338779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/How%20much%20Ram%20do%20we%20need%3F/near/235338779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/How.20much.20Ram.20do.20we.20need.3F.html#235338779">(Apr 20 2021 at 12:58)</a>:</h4>
<p>so instead of outright deleting a value it just hands out a reference</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>